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BACKGROUND OF TH K INVENTION 

1 . Field of the Invention 

The invention relates to methods and apparatus 
for processing multimedia messages. More particularly, 
the invention relates to methods and apparatus for 

(1) converting messages from one medium to another; 

(2) performing message content analysis; (3) utlizizing 
linguistically based analysis tools to identify message 
relationships regardless of media type; (4) interrelating 
messages according. to content; and (5) providing a simple 
message reference capability to simplify message access. 

2 . Brief Description of the "Prior Art 

Business people receive many different kinds of 

messages, e.g. electronic mail, voice mail, fax, video 

messages, attachments to electronic mail. It is possible 
and desirable to have all messages sent to a single mail 
box from which they may all be retrieved regardless of 
the message type. However, the only retrieval device 
which is capable of reading all of these different types 
of messages is a personal computer having a graphical 

-display and audio video capability. Unfortunately, it is 



not always possible or convenient -to retrieve messages 
with a personal computer. 



A unified mailbox where all kinds of media 
5 (voice, fax, e-mail, and video) are made accessible 

and/or visible from virtually anywhere to a subscriber or 
user in one basket is a convenient means of communication 
when compared to handling multiple mailboxes with 
distinct media. Current solutions for a unified mailbox 
10 are inefficient, however, for someone with an intense 

communication style and a frequent need to handle his/her 
messages remotely. The mismatch of media type of the 
information and the capabilities of the various (often 
limited) devices used for remote access places a heavy 
15 burden on the user and the interface of the system. This 
is especially true for the interfaces utilizing a 
telephone with no display, or handheld devices with 
limited display capabilities. 

20 Some of the problems arise in the context of 

compound and/or lengthy messages in connection with one 
or the other access means. For example, it is not 
possible to deliver voice and fax messages to a text-only 
e-mail capable device. It is also difficult to deal with 

25 lengthy e-mails delivered to a voice-only interface or to 
a text-interface with limited capabilities. Even when 
the device has a fully functional GUI interface, there is 
room for increased efficiency with large amounts of data. 
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It is a challenge to eff iciently- present the " information 
in various office document formats (e.g., Word Processor, 
Spreadsheet, and Presentations) associated with a 
message. It is often difficult to locate and visually 
5 present related messages and attachments. When the 
mailbox has many messages in it, it is difficult to 
reference the messages . 

Other problems arise due^ to the -increased 
10 amount of information the unified mailbox can provide. 
Current mechanisms for organizing and presenting 
relationships among messages (listing by arrival time, 
subject, sender, etc.) are insufficient for a large 
number of messages of varying media and, especially, 
15 mixed media within a given message. 

It would be desirable to provide a flexible, 
media independent way of finding and navigating related 
messages. With current systems, for example, the user is 
20 unable to recognize that there is a relationship between 
a voice message and a fax without listening to the voice 
message and displaying/printing the fax. 

Because the presentation of unified mailbox 
25 information is more complex, especially if relationships 
as described hereinabove are incorporated into the 
presentation, identifying an individual item (message or 
message attachment) for further action can become 
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problematic. How does the client/user identify to the 
server which message is to be acted upon? Are the entire 
message and its attachments to be involved? Is it a 
single attachment or only the original message body? And 
if the messages are presented in a "graph" format, how 
does the user select an individual item? 

Current unified mailbox systems offer media 
sensitivity for message retrieval only when -accessed with 
a graphical user interface (GUI) from a PC client. If a 
particular media or office document is attached to an 
e-mail, the user needs to click-on it in. order to launch 
a specific application, for example, an audio player for 
voice, tiff-viewer for fax, video player to view a video 
15 message, etc. 

For users with intense communication 
requirements (e.g. executives or customer service agents 
who receive hundreds of compound messages daily) there 
are no means to quickly process inbox messages except by 
the sender information, the subject line, and maybe few 
lines of the message body. In order to read messages, 
the user has to click on or mark a certain item in a 
graphical interface in order to get to the message body. 



20 



25 



No content summarization of lengthy text 
messages or respective attachments is available yet that 
would remarkably improve the efficiency of handling- the 
daily information avalanche in the office. 
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No 



15 



Current mailbox searching does not provide 
visual display of content and temporal relationships, 
search capability exists yet for non-text messages. 

If a unified mailbox is accessed from a 
telephone interface, voice and e-mail messages are 
retrievable and the user can listen to both. Existing 
text-to-speech technology provides a means-to-convert the 
e-mail to voice. A fax message can be forwarded to a fax 
machine or printer. 

However, if an e-mail contains an attachment, 
the systems are able to indicate that, but are ^unable to 
access its content. Similarly, the contents of a fax or 
other documents attached to an e-mail are indicated but 
not accessible to the user accessing the mailbox with a 



telephone interlace. 



20 



If an e-mail is lengthy, the user may be able 
to navigate through it by accelerating the text-to-speech 
reading speed. However, there is no means of text 
content summarization applied to shorten the process 
without eventually losing/skipping critical content. 
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If messages are forwarded to a handheld device 
via a wireless service but the device has limited 
text-display capabilities only certain parts of the email 

5 (From, Subject and a limited number of characters of the 
message body) can be displayed. If the critical 
information in the message is not in the beginning of the 
message body that is displayed, it is "lost" to the 
recipient. He/she has to use other access-methods or 

10 make a call into the messaging system/ server to retrieve 
the full text message (by listening to it or by 
initiating a printing to a device nearby) . 

As mentioned above, voice and other media 
15 attachments are indicated but not transmitted and/or 

displayed on a text-only display. The user needs to use 
other access methods to retrieve the messages. 
Additionally, no text content summarization methods are 
utilized to deal with access " device technology 
20 limitations. 

Full message sensitivity is only provided when 
accessing a mailbox with a multimedia PC. However even 
multimedia PCs lack any means to summarize message 
25 content in order to make it more efficient for the 

recipient to read his/her .lengthy messages. Also, there 
are yet no means to summarize content of attached 
documents . 
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When accessing a mailbox with a telephone, the 
media and device sensitivity is limited to voice and 
e-mail. Again, no techniques of text content 
summarization are applied yet in order to make the 
retrieval of the message information over the phone more 
convenient. 

in the case of handheld or mobile devices with 
limited text-display capabilities, the problem is that 
lengthy messages are usually not transmitted in their 
entirety by the wireless/paging service providers. 
Additionally, any other media attachments are "lost". No 
content summarization of lengthy text messages or 
respective attachments is available yet that would 
remarkably improve the efficiency of handling the daily 
information avalanche in the office. 
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SUMMARY OF THE INVENTION 



It is therefore an object of the invention to 
provide methods and apparatus for accessing multimedia 
messages from a unified mailbox. 

It is also an object of the invention, to 
provide methods and apparatus for converting media types 
in a unified multimedia mailbox. 

It is another object of the invention to 
provide methods and apparatus for summarizing the content 
of messages in a unified multimedia mailbox. 

It is yet another object of the invention to 
provide methods and apparatus for cross referencing 
related messages based on content. 

It is another object of the invention to 
provide methods and apparatus for improved handling of 
email attachments . 

It is still another object of the invention to 
provide methods and apparatus for customizing mail 
handling based on a system profile adapted to the device 
used to access the mailbox. 
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In accord with these objects which will be 
discussed in detail below the apparatus and associated 
methods of the invention include a mail server that 
provides multimedia message inbox for one or several 
5 users on a network; a subsystem that detects media 

attachments to messages in a mailbox; a subsystem that 
converts media attachments into another media type using 
text-to-speech, fax-to-text, video voice track into text 
and speech-to-text; a subsystem that analyzes and 
10 summarizes the text content of original or converted 

media in respect of the linguistic meaning; a subsystem 
that delivers appropriate media according to an access 
device and message purpose, as defined in a profile; a 
, subsystem that identifies cross-media interrelationships 
15 between messages and controls the media conversions 
necessary for this analysis; and a subsystem that 
controls a reference number scheme. 

The methods and apparatus of the invention 
20 solve the problems discussed above by utilizing advanced 
media conversion methods, analysis and summarization of 
message content, and intelligent forwarding concepts. It 
provides access device and media sensitive intelligence 
for a mailbox when retrieving or forwarding a particular 
25 message. 
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The concept of media conversion is extended beyond 
text-to-speech to other attachments; a 

speaker-independent, large vocabulary, telephony-quality 
speech recognition engine is utilized to convert a voice 
message to text or to convert the voice track of a video 
attachment into readable text. Similarly, fax 
information is converted into text. 

According to the invention, the content of 
messages is automatically summarized. The summarization 
of a message content is an improvement toward efficiency, 
particularly in the case of a forwarded lengthy message 
to a handheld device with limited display capabilities. 
The same is true for reading a lengthy message over the 
phone. Summarization is also applied to attached media . 
(e.g. fax, Word document) extends even the media content 

dCCcSSiuic .' 

Both, the media conversion and the content 
summarization applied together provide compatibility with 
the access device. Depending on the user, the types of 
potential access devices are usually predefined; 
therefore messages along with their attachments that form 
the message content can be tailored to those devices 
while accessed or forwarded according to a profile. This 
ensures the availability of more information to the 
recipient at the device of choice and that is probably 
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most convenient. Still, if the user requires more 
information, he/she can utilize another access method. 



The invention also provides cross-media 
searching and visual displaying. often messages related 
to a specific topic of interest to the user are in 
different media and spread throughout the message store 
(e.g. in different folders). The cross-media search 
finds these messages and presents them to the user in a 
way that makes the content and time relationships clear 
allowing efficient use of the otherwise overwhelming 
amount of information. The search engine utilizes 
sophisticated linguistically based analysis tools to 
discover the message relationships. 

Additionally, a reference number scheme for all. 
messages is provided. All messages in a particular group 
of messages of interest to the user are assignee a 
reference number to be used in further actions. Thus a 
PDA user can, for example, get a summary of messages with 
reference numbers and an indication of the message type. 
This, reference number may then be used to access that 
message, and through it, a particular attachment to that 
message for further. Voice commands may be used to 
invoke actions on items more efficiently using the 
reference numbers of messages. 
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RF.TEF DES^PTPTTON OF THE DRAWINGS 

FIG. 1 is a high level block diagram of a 
multimedia mail system according to the invention. 
5 . 

nFTATT.ED nFfiCRIPTION 

Turning now to Figure 1, an integrated 
10 multimedia messaging system according to the -invention 
includes a mail server 10 that provides multimedia 
message inbox for one or several users on a network; a 
mail processor 11; a subsystem 12 that detects media 
attachments to messages in a mailbox; one or more 
15 subsystems that converts media attachments into another 
media type using text-to-speech 14, fax-to-text 16, video 
voice track into text 18 and speech-to-text 20 a 
subsystem 22 that analyzes and summarizes the text 

rocnar-t Cf the 



content of original c — * 

20 linguistic meaning; a subsystem 24 that delivers 

appropriate media according to an access device and 
message purpose, as defined in a profile; a subsystem 26 
that identifies cross-media interrelationships between 
messages and controls the media conversions necessary for 
25 this analysis; and a subsystem 28 that controls a 
reference number scheme. 
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The invention can better be understood through 
an illustrative example such as the notification of a 
single-media voice message to a data pager. The 
5 following describes an example of this process involving 
a user that has a multimedia mailbox and a data pager who 
receives a voice message. 



The problem is to provide the "best !'-...,„, - - 
information to the pager so the user can proceed most 
efficiently. What is the "best" information will vary 
according to the user's actual preferences, but will most 
likely include sender identification and meaningful 
portions of the message itself. In addition, there are 
probably messages the user would prefer to delay any 
handling of until an appropriate device is available. 
Thus the steps for sending voice messages to a pager 
would include: a) filtering messages to be processed, 
b) speech-to-text conversion, c) summarization and post 
filtering, and d) selection and delivery of text 
information to the device. 



15 



20 



25 



Since the resources involved in processing a 
message may be large, messages are pre-f iltered . 
Speech-to-Text is "expensive" in its use of resources, 
interrupting the user with any but the most important 
messages can be an unnecessary expense of the user's time 
and attention as well as a waste of system resources. 
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Thus a mechanism to prevent the presentation of a message 
to a given device is important. This filtering is based 
on a variety of data including sender, message priority, 
etc. and the criteria for filtering is stored in the 
5 system profile for the user. 

Voice messages which pass through the pre- 
filter are converted to text. This is most efficiently 
accomplished on the server side, perhaps with a dedicated 
10 "helper" server explicitly for the server so as not to 
disturb other processing on the server. The resulting 
text message is then be associated with the original 
message (as the text message body or as a separate 
attachment) . 

15 

Before sending the text message to the pager, 
it is subjected to post-conversion filtering and 
summarization. Post-conversion filtering is optional, 
preventing processing of messages that appear not to be 
20 on a topic deemed important to the user. If it does not 
appear important, it would then remain in the mailbox to 
be processed. If the message survives the post- 
conversion filtering step, the text is then summarized. 

25 Most simply, summarization includes reduction 

to a list of keywords and phrases found within the text. 
The summarization is created by removing from the message 
words/phrases not found within the user-defined list of 
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keywords/phrases. More complex summarization includes 
allowing the user to specify the keyword/phrase list 
based on the sender of the message. 

Since the message is a speech-to-text 
conversion, the keywords and their homonyms should be 
checked. An option on the summarization, for example a 
check box that says - allow homonyms", could be utilized 
to enable this feature. _ ~ 

Even more complex summarization methods 
contemplated by the invention involve performing 
sophisticated grammatical parsing and analysis. 

Data is transmitted to the pager based on a 
user defined data selection criteria which is stored as a 
template in the system profile for the user. The data 
available for selection includes sender «*., 
summary, message priority, un-summarized text, and other 
20 fields as available. 

The user describes a template that indicates 
the information desired and the number of 
characters of each field desired. For example: 

•-From %SENDER% at %TIME%: %100SUMMARY% " 
indicates that the user wants a string that includes the 
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15 



entire sender name, the received time and the first 100 
characters of the summary to appear on his pager. 

When the user receives the page, the summary 
5 information gives him/her enough information to determine 
how critical the message is. If it appears critical, 
he/she may choose to access the entire message using a 
different device, e.g. a telephone. 

10 Another example is the retrieval of text 

messages (such as email) via a telephone. Text messages 
are pre-filtered as described above. The text is then 
summarized. The summary is then converted to speech 
which is played on the telephone to the user calling in 

15 for messages. 

Still another example is sending a fax message 
^ _ rj0 7\ .--»w -.^cc^^oc a >-e ore- filtered based on sender 
and priority. The fax messages which pass through the 
filter are converted to text with OCR (optical character 
recognition) software. The text is summarized. Data is 
selected using a user defined template. The text message 
is sent tot he PDA and the user is notified 
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1 1 



In general, a user can define a morphing 
process 1 • for messages in the context of any particular 
target device such as a pager or a cell phone with a 
limited display. 
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The morphing process is a combination of 
message filtering, message restructuring, data 
conversion, data summarization, data selection and 
5 notification steps that are configured to handle 

particular media types for particular target devices. 
Each user may define a set of rules and parameters for 
each device type defining how messages are morphed. 

10 For example, a user may have a Voice 

Message-to-Pager morph definition that would do the 
following: 

(a) filter messages based on sender and priority, 
IS removing from further processing (i.e. leaving 

on the server) messages that are not deemed 
urgent enough to disturb the user while out of 
the office; 

20 (b) perform speech-to-text conversion ; 

(c) summarize the text based on criteria defined by 
the user; 

25 (d) perform further filtering based on the 

summarized/converted text; 

(e) organize the text in a template; and 
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(f) send the message to the pager. 



In general, a morphing process will include 
these steps in some order determined by the user. In 
addition, message restructuring steps allow the user to 
handle multiple attachments of varying media attached to 
the message. For example, the user may select that a 
summary of the attachments be created (attachment name 
and media type) or may request that the attachments be 
expanded, converted and summarized* as described above for 
the single media message. 

There have been described and illustrated 
herein methods and apparatus for processing multimedia 
messages. While particular embodiments of the invention 
have been described, it is not intended that the 
invention be limited thereto, as it is intended that the 
invention be as broad in scope as the art will allow and 
that the specification be read likewise. It will 
therefore be appreciated by those skilled in the art that 
yet other modifications could be made to the provided 
invention without deviating from its spirit and scope as 
so claimed. 
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What is claimed is: 

1. A multimedia mailbox system, comprising: 
5 a) a message store for storing multimedia messages; and 

b) a plurality of data converters for converting 
messages in one medium to messages in another 
medium. 
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2. A system according to claim 1 wherein said plurality 
of data converters includes at least two selected from 
the group consisting of a text to speech converter , a 
speech to text converter, and a fax to text converter. 



3. A system according to claim 1 further comprising 
means for linguistically based searching of multiple 
20 message types and for linguistically relating multiple 
messages of different type. 



4. A system according to claim 1 further comprising 
25 means for assigning a reference number to each message. 
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5. A multimedia mailbox system, comprising: 

a) a message store for storing multimedia messages; and 

b) means for linguistically based searching of multiple 
message types and for linguistically relating 
multiple messages of different type. 

6. A system according to claim 5 further comprising 
means for assigning a reference number to each message. 

7. A multimedia mailbox system, comprising: 

a) a message store for storing multimedia messages; 

b) a plurality of data converters for converting 
messages in one medium to messages in another 
medium; 

c) means for linguistically based searching of multiple 
message types and for linguistically relating 
multiple messages of different type and 

d) means for assigning a reference number to each 
message . 
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8. A system according to claim 7 wherein said plurality 
of data converters includes at least two selected from 
the group consisting of a text to speech converter, a 
5 speech to text converter, and a fax to text converter. 



9. A method for managing a multimedia mailbox, 
comprising the steps of: 



10 



a) storing messages of different types in a single 
mailbox; and 

b) automatically converting messages from one medium to 

15 another. 



10. A method according to claim 9 wherein said step of 
converting includes at least- two selected from the group 
20 consisting of converting text to speech, converting 
speech to text, and converting fax to text. 



11. A method according to claim 9 further comprising the 
25 step of linguistically searching multiple message types 
and linguistically relating multiple messages of 
different type. 
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12. A method according to claim 9 further comprising the 
step of assigning a reference number to each message. 



5 13. A method for managing a multimedia mailbox, 
comprising the steps of: 

a) storing messages of different types in a single 
mailbox; and 

10 

b) linguistically searching multiple message types and 
linguistically relating multiple messages of 
different type. 



15 

14. A method according to claim 13 further comprising the. 
step of assigning a reference number to each message. 
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15. A method for managing a multimedia mailbox, 
comprising the steps of: 

5 a) storing messages of different types in a single 
mailbox; 

b) automatically converting messages from one medium to 
another; 

10 

c) linguistically searching multiple message types and 
linguistically relating multiple messages of 
different, type; and 

15 d) assigning a reference number to each message. 



16. A method according to claim 15 wherein saia ^ 
converting includes at least- two selected from the group 
20 consisting of converting text to speech, converting 
speech to text, and converting fax to text. 
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ABSTRACT OF THE DISC LOSURE 

The invention provides the user of a unified 
messaging mailbox with efficient, intelligent, media and 
5 device sensitive methods and apparatus to access and 
process (e.g., read, listen, forward, and search) 
messages. The invention introduces media conversion 
capabilities to selectively treat multimedia messages and 
message attachments so that they can b^ . efficiently 
10 handled by mobile devices like PDAs (Personal Digital 

Assistants), pagers, or phone devices (with or without a 
text display feature) . Furthermore, the invention 
introduces message content analysis capabilities that 
will recognize linguistic relationships between messages 
15 regardless of the media type. The invention also 
describes the ability to present these linguistic 
relationships along with the standard messaging 
relationships (Message arrival time, subject, sender, 
etc.). Still further, the invention introduces a message 
20 referencing option that allows simpler message selection 
from certain devices. 
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ASSIGNMENT 

For good and valuable consideration, we, Christoph A. Aktas residing at 992 
Planetree, Sunnyvale, CA 94086, citizen of Germany, John W. Yates residing at 
106A Kittoe Drive, Mountain View, CA 94043, citizen of USA, and Phillip C 
Meredith residing at 459 Homer Avenue #6, Palo Alto, CA 94301, citizen of USA 

Hereby sell, assign, and transfer to SIEMENS INFORMATION AND 
COMMUNICATION NETWORKS, INC., a corporation of the State of Delaware, 
having a principal place of business at 900 Broken Sound Parkway, Boca Raton, FL 
33487, hereinafter "Assignee 11 , its successors, assigns and legal representatives, the entire 
right, title and interest in and for the United States and all foreign countries, in and to any 
and all improvements which are disclosed in the application for United States Letters 
Patent, which has been executed by the undersigned concurrently herewith, and is 
entitled: 

METHODS AND APPARATUS FOR ACCESSING AND PROCESSING 
MULTIMEDIA MESSAGES STORED IN A UNIFIED MULTIMEDIA 

MAILBOX 

and in and to said application and all divisional, continuing, substitute, renewal, reissue, 
and all other applications for Letters Patent which have been or shall be filed in the 
United States and all foreign countries on any of said improvements; and in and to all 
original and reissued patents which have been or shall be filed in the United States and 
all foreign countries on said improvements; 

Agree that said Assignee may apply for and receive Letters Patent for said 
improvements in its own name; and that, when requested, without charge to but at the 
expense of said Assignee, its successors, assigns and legal representatives, to carry out 
in good faith the intent and purpose of this assignment, the undersigned will execute 
all divisional, continuing, substitute, renewal, reissue, and all other patent applications 
on any and all said improvements; execute all rightful oaths, assignments, powers of 
attorney and other papers; communicate to said Assignee, its successors, assigns, and 
legal representatives, all facts known to the undersigned relating to said improvements 
and the history thereof; and generally do everything possible which said Assignee, its 
successors, assigns or legal representatives shall consider desirable for aiding in 
securing and maintaining proper patent protection for said improvements and for 
vesting title to said improvements and all applications for patents and all patents on 
said improvements, in said Assignee, its successors, assigns and legal representatives; 
and 
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Covenant with said Assignee, its successors, assigns and legal representatives that 
no assignment, grant, mortgage, license or other agreement affecting the rights and 
property herein conveyed has been made to others by the undersigned, and that full nght 
to convey the same as herein expressed is possessed by the undersigned. 



ACKNOWLEDGEMENTS 




Christoph A. Aktas 

Witness No. 1 : 

Rv: /^L (% Date: i / / 

Printed Name: f'i)* O ^e^i-r C kcA± 



Witness No. 2: rX-- „ „ 

By: 



Printed Name: 3>r. U)oif^<^<i 1>ckrrt 



Date: v w / 



John W. Yates 

Witness No. 1: 

By: Date: 

Printed Name: 



Witness No. 2: 

By:_ Date- 
Printed Name: . 
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Covenant with said Assignee, its successors, assigns and legal representatives that 
no assignment, grant, mortgage, license or other agreement affecting the rights and 
property herein conveyed has been made to others by the undersigned, and that full right 
to convey the same as herein expressed is possessed by the undersigned. 



a r k n o w t f n r, f. m h n t s 



Christoph A. Aktas 



Witness No. 1: 



By: 

Printed Name: 



Date: 



Witness No. 2: 



By: 

Printed Name: 



Date: 



Jojin W. Yates \ 



Witness No. 1: 

By: f .^vw - . Date: .J 2.oot 

Printed Name / //'.•?--✓ &>;., Hm-,.- 

Witness No. 2: 

Date: 3 / Z>/p / 
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Phillip C. Meredith 

Witness No. 1: 

Bv ^l2^/zCL r <= ^fg^7 ^ Date: ^3 / * ( 

Printed Name: ^f? odiHzrru). bc/rc^Y^ 



Witness No. 2: 

Bv: P^-^is U^.Lu\ ^l^UASt^Ot ^ Date: j~3" 
Printed Name: i^b,>> ?Krkr WAjt^etUM^ 
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